Transfer Learning: From a Translation Model to a Dense Sentence Representation with Application to Paraphrase Detection

It is becoming increasingly difficult to judge the originality of digital content, especially with the continuously growing corpus of digital text. Traditional methods such as n-gram overlap are susceptible to simple obfuscation techniques. In this paper, we investigate several methods of measuring sentence similarity, with particular emphasis on paraphrase detection. We show that a bag-of-vectors approach provides a simple yet effective baseline. We then demonstrate how the encoder from a neural machine translation model can be used to build a powerful paraphrase detection classifier. Duplication of digital text, whether accidentally or deliberately copied, is a major concern for the academic and business communities, alike. Traditional plagiarism detection methods, such as ngram overlap, are susceptible to simple obfuscation techniques (Potthast et al., 2010). A more robust method for comparing the semantic content of sentences is needed. Recently, researchers have shown that using dense representations of words and sentences, along with deep learning techniques, can provide impressive results on a number of natural language processing (NLP) tasks (Socher et al., 2011; Wu et al., 2016). A major roadblock in the development of paraphrase classification models is the lack of large, highquality datasets. Most of the large paraphrase datasets have less than 20,000 sentence pairs (Dolan and Brockett, 2005; Xu et al., 2014). While sufficient for traditional machine learning techniques, the publicly available paraphrase datasets are not sufficiently large to train complex deep learning models. The training difficulty arises because accurate paraphrase detection requires a certain level of semantic understanding. One solution is to apply transfer learning, where a natural language model is pretrained on a separate task, before being applied to the target task (Pan and Yang, 2010). Transfer learning has seen many applications in image processing, where the lower layers of a pretrained model are often used to initialize a new model (Oquab et al., 2014). In some cases the weights of the lower layers can be fixed while training the second model. A similar approach is applied in (Pan et al., 2010) where a denoising autoencoder is trained on a large dataset and then used for a sentiment classification task. In this work, we demonstrate how the encoder from a neural machine translation (NMT) model can be used to build a powerful paraphrase detection classifier. We start by constructing a simple bagof-words baseline for the paraphrase detection task. We then demonstrate that the context vector from a translation vector can be used to classify paraphrases with greater accuracy than the baseline. Finally, we combine the baseline model with the NMT model, to generate an ensemble model which approaches state-of-the-art performance.